{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set up a Python virtual environment in Visual Studio Code\n",
    "\n",
    "1. Open the Command Palette (Ctrl+Shift+P).\n",
    "1. Search for **Python: Create Environment**.\n",
    "1. Select **Venv**.\n",
    "1. Select a Python interpreter. Choose 3.10 or later.\n",
    "\n",
    "It can take a minute to set up. If you run into problems, see [Python environments in VS Code](https://code.visualstudio.com/docs/python/environments)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "vscode": {
     "languageId": "plaintext"
    }
   },
   "source": [
    "### Install packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "! pip install -r phi-chat-requirements.txt --quiet"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Provision the sample\n",
    "\n",
    "This sample uses [`azd`](https://learn.microsoft.com/azure/developer/azure-developer-cli/), a bicep template, and a custom post-provision hook to provision the sample. The sample uses role-based authentication using your identity for authentication. API keys are used to authenticate to the Phi 3 endpoint.\n",
    "\n",
    "1. Open a PowerShell command prompt in the phi-chat folder.\n",
    "\n",
    "1. Run `azd config set defaults.subscription <yourSubscriptionID>` to set the subscription if you have multiple Azure subscriptions.\n",
    "1. Run `azd env new <your-environment-name>` to create a new `azd` environment. This environment will contain a set of environment variables used to provision the sample.\n",
    "1. If you want to use an existing resource, set the corresponding `azd` environment variables before deployment:\n",
    "   1. Existing Search resource:\n",
    "      1. `azd env set AZURE_SEARCH_SERVICE <your-search-service-name>`\n",
    "      1. `azd env set AZURE_SEARCH_SERVICE_LOCATION <your-search-service-location>`\n",
    "      1. `azd env set AZURE_SEARCH_SERVICE_RESOURCE_GROUP <your-search-service-resource-group>`\n",
    "      1. `azd env set AZURE_SEARCH_SERVICE_SKU <your-search-service-sku>`\n",
    "      1. `azd env set AZURE_SEARCH_SERVICE_SEMANTIC_RANKER <your-semantic-ranker-sku>`\n",
    "   1. Existing Azure OpenAI account:\n",
    "      1. `azd env set AZURE_OPENAI_ACCOUNT <your-openai-account-name>`\n",
    "      1. `azd env set AZURE_OPENAI_LOCATION <your-openai-account-location>`\n",
    "      1. `azd env set AZURE_OPENAI_RESOURCE_GROUP <your-openai-resource-group>`\n",
    "   1. By default, `text-embedding-3-large` with 3072 dimensions is used to embed the sample documents. You can change this by setting the following environment variables:\n",
    "      1. `azd env set AZURE_OPENAI_EMB_MODEL_NAME <embedding-model-name>`\n",
    "      1. `azd env set AZURE_OPENAI_EMB_MODEL_VERSION <embedding-model-version>`\n",
    "      1. `azd env set AZURE_OPENAI_EMB_MODEL_DIMENSIONS <embedding-model-dimensions>`\n",
    "      1. `azd env set AZURE_OPENAI_EMB_DEPLOYMENT <embedding-model-deployment-name>`\n",
    "      1. `azd env set AZURE_OPENAI_EMB_DEPLOYMENT_CAPACITY <embedding-model-deployment-capacity>`\n",
    "   1. By default, gpt-35-turbo and gpt-4 are deployed alongside Phi 3 for comparison. You can disable and configure these deployments using the following environment variables:\n",
    "      1. `azd env set ENABLE_GPT35TURBO false`\n",
    "      1. `azd env set ENABLE_GPT4 false`\n",
    "   1. You can use an existing AI Hub, AI Project, Online Endpoint, or deployment by setting the following environment variables:\n",
    "      1. `azd env set AZUREAI_HUB_NAME <ai-hub-name>`\n",
    "      1. `azd env set AZUREAI_PROJECT_NAME <ai-project-name>`\n",
    "      1. `azd env set AZUREAI_ONLINE_ENDPOINT_NAME <ai-online-endpoint-name>`\n",
    "      1. `azd env set AZUREAI_DEPLOYMENT_NAME <ai-deployment-name>`\n",
    "   1. You can change which model is deployed by pointing to a different model on the [AzureML Registry](https://learn.microsoft.com/azure/machine-learning/how-to-manage-models) by setting the following environment variable:\n",
    "      1. `azd env set AZURE_DEPLOYMENT_MODEL <azureml://registries/registry/models/model/versions/version`\n",
    "   1. You can change the compute type and deployment capacity by setting the following environment variables:\n",
    "      1. `azd env set AZURE_DEPLOYMENT_INSTANCE_TYPE <compute-instance-type>`\n",
    "      1. `azd env set AZURE_DEPLOYMENT_CAPACITY <compute-capacity>`\n",
    "1. Run `azd provision`.\n",
    "   1. Enter a development environment name.\n",
    "   1. Enter a region for the deployment. Be sure to choose a region you have capacity for a Phi 3 deployment in.\n",
    "\n",
    "This step may take up to a half an hour to complete. Provisioning compute for Phi 3 may take a long time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Retrieve environment variables after provisioning\n",
    "\n",
    "The included `azd` bicep template saves all required environment variables for the notebook automatically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Load all environment variables from the azd deployment\n",
    "import subprocess\n",
    "from io import StringIO\n",
    "from dotenv import load_dotenv\n",
    "import os\n",
    "result = subprocess.run([\"azd\", \"env\", \"get-values\"], stdout=subprocess.PIPE, cwd=os.getcwd())\n",
    "load_dotenv(override=True, stream=StringIO(result.stdout.decode(\"utf-8\")))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Validate the indexer has completed successfully\n",
    "\n",
    "An indexer runs in the background to chunk and vectorize all the sample documents. Validate that it has completed without any errors before trying to search the sample index. If there are any errors, they may be due to temporary throttling from the Azure OpenAI embeddings. Try re-running the indexer to resolve this issue."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Status: success\n"
     ]
    }
   ],
   "source": [
    "from azure.search.documents.indexes import SearchIndexerClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "search_indexer_client = SearchIndexerClient(endpoint=os.getenv(\"AZURE_SEARCH_ENDPOINT\"), credential=DefaultAzureCredential())\n",
    "status = search_indexer_client.get_indexer_status(name=os.getenv(\"AZURE_SEARCH_INDEXER\"))\n",
    "print(f\"Status: {status.last_result.status}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get Phi 3 scoring URI and key\n",
    "\n",
    "Connect to the Azure AI Project and fetch the deployed Phi scoring uri and key for access"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import MLClient\n",
    "from azure.identity import DefaultAzureCredential\n",
    "\n",
    "workspace_ml_client = MLClient(\n",
    "    DefaultAzureCredential(),\n",
    "    subscription_id=os.getenv(\"AZURE_SUBSCRIPTION_ID\"),\n",
    "    resource_group_name=os.getenv(\"AZURE_RESOURCE_GROUP\"),\n",
    "    workspace_name=os.getenv(\"AZUREAI_PROJECT_NAME\")\n",
    ")\n",
    "\n",
    "phi3_url = workspace_ml_client.online_endpoints.get(name=os.getenv(\"AZUREAI_ONLINE_ENDPOINT_NAME\")).scoring_uri\n",
    "phi3_key = workspace_ml_client.online_endpoints.get_keys(name=os.getenv(\"AZUREAI_ONLINE_ENDPOINT_NAME\")).primary_key"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup search parameters for grounding the conversation on your documents\n",
    "\n",
    "Change the various types of searches to change how chunks are fetched from the index.\n",
    "\n",
    "* `k` - Determines how many documents are fetched by either the vector search step and the text portion of the hybrid search step. This parameter is used to improve search result quality.\n",
    "* `search_type` - `text` for plain text search, `vector` for plain semantic similiarity search using an OpenAI embedding model, or `hybrid` to combine both text and semantic similarity search using Reciprocal Rank Fusion (RRF).\n",
    "* `use_semantic_reranker` - Whether or not to rerank the top results using the semantic reranker model for improved accuracy.\n",
    "* `sources_to_include` - How many documents to send to the Language Model for RAG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "k=50\n",
    "search_type=\"hybrid\"\n",
    "use_semantic_reranker=True\n",
    "sources_to_include=5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Chat using Phi 3\n",
    "\n",
    "Start the conversation using Phi 3 using a minimal RAG implementation located in [lib/chat.py](./lib/chat.py). To increase the amount of tokens Phi 3 is allowed to respond with, change the `max_new_tokens` parameter in `get_phi3_response` to a larger number. Note that the sample deploys the Phi 3 version with a 4k context limit by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Northwind Health Plus includes coverage for mental health and substance abuse treatments, unlike Northwind Standard. \n",
      "\n",
      "Northwind Standard does not cover mental health and substance abuse treatments. Northwind Health Plus offers a comprehensive plan that includes emergency services, mental health and substance abuse coverage, and out-of-network services, which are not covered by Northwind Standard. Additionally, Northwind Health Plus provides more prescription drug coverage compared to Northwind Standard.\n",
      "\n",
      "If you require emergency services, ensure your provider is part of the Northwind Health network to avoid full cost responsibility.\n",
      "\n",
      "For mental health and substance abuse treatments, verify that your provider is networked with Northwind Health Plus.\n",
      "\n",
      "Coordinate your benefits with other health care plans if you have additional coverage.\n",
      "\n",
      "Northwind Health Plus covers emergency services, mental health and substance abuse treatments, and out-of-network\n"
     ]
    }
   ],
   "source": [
    "import azure.identity.aio\n",
    "from lib.chat import ChatThread, create_search_client, SearchType\n",
    "\n",
    "chat_thread = ChatThread()\n",
    "\n",
    "async with azure.identity.aio.DefaultAzureCredential() as credential, create_search_client(credential) as search_client:\n",
    "    await chat_thread.append_grounded_message(\n",
    "        search_client=search_client,\n",
    "        query=\"What is included in my Northwind Health Plus plan that is not in standard?\",\n",
    "        search_type=SearchType(search_type),\n",
    "        use_semantic_reranker=use_semantic_reranker,\n",
    "        sources_to_include=sources_to_include,\n",
    "        k=k)\n",
    "    await chat_thread.get_phi3_response(\n",
    "        endpoint_scoring_uri=phi3_url,\n",
    "        endpoint_authorization=\"Bearer \" + phi3_key,\n",
    "        deployment=os.getenv(\"AZUREAI_DEPLOYMENT_NAME\"),\n",
    "        max_new_tokens=200)\n",
    "\n",
    "print(chat_thread.get_last_message()[\"content\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## (Optional) Chat using GPT4 or GPT-35-Turbo\n",
    "\n",
    "Continue your conversation using OpenAI models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "- Coverage for hospital stays, X-rays, and a broader range of prescription drugs not included in Northwind Standard.\n",
      "- Access to both in-network and out-of-network emergency services.\n",
      "- Personal health support programs, including case management, disease management, and wellness programs exclusive to Northwind Health Plus.\n",
      "- Northwind Health Plus includes services by independent contractors, if medically necessary.\n",
      "- Offers mental health and substance abuse coverage, which are also included in Northwind Standard.\n",
      "- Coverage for out-of-network services, unlike Northwind Standard.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from lib.chat import ChatThread, create_search_client, create_openai_client, SearchType\n",
    "\n",
    "chat_deployment = os.getenv(\"AZURE_OPENAI_GPT4_DEPLOYMENT_NAME\") # replace with AZURE_OPENAI_GPT35TURBO_DEPLOYMENT_NAME to use gpt-35-turbo\n",
    "\n",
    "async with azure.identity.aio.DefaultAzureCredential() as credential, create_search_client(credential) as search_client, create_openai_client(credential) as openai_client:\n",
    "    await chat_thread.append_grounded_message(\n",
    "        search_client=search_client,\n",
    "        query=\"What is included in my Northwind Health Plus plan that is not in standard?\",\n",
    "        search_type=SearchType(search_type),\n",
    "        use_semantic_reranker=use_semantic_reranker,\n",
    "        sources_to_include=sources_to_include,\n",
    "        k=k)\n",
    "    await chat_thread.get_openai_response(openai_client=openai_client, model=chat_deployment)\n",
    "\n",
    "print(chat_thread.get_last_message()[\"content\"])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Inspect sources used by the Language Model\n",
    "\n",
    "Check which sources were used to ground the response"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'content': 'includes counselling, psychotherapy, \\n\\nand other treatments related to mental health and substance abuse.  \\n\\n\\n\\nOut-of-Network Services: The Northwind Standard plan does not cover any services that are \\n\\nprovided by a provider that is not part of the Northwind Health network. This includes \\n\\ndoctors, hospitals, and other healthcare providers who are not part of the Northwind Health \\n\\nnetwork.  \\n\\nTips \\n\\nWhen selecting a healthcare plan, it is important to be aware of the exclusions in the plan. \\n\\nHere are some tips to help you understand the exclusions in the Northwind Standard plan:  \\n\\n1. Understand the types of services that are not covered by the Northwind Standard plan. Be \\n\\nsure to familiarize yourself with the list of exclusions and make sure that any services you \\n\\nmight require are covered.  \\n\\n2. If you require emergency services, be sure to check with your provider to see if they are \\n\\npart of the Northwind Health network. If they are not, you will be responsible for the full \\n\\ncost of those services.  \\n\\n3. If you require mental health or substance abuse treatments, be sure to check with your \\n\\nprovider to see if they are part of the Northwind Health network. These services are not \\n\\ncovered by the Northwind Standard plan.  \\n\\n4. If you require services from a provider that is not part of the Northwind Health network, \\n\\nyou will be responsible for the full cost of those services.  \\n\\nBy understanding the exclusions in the Northwind Standard plan, you can make informed \\n\\ndecisions about your healthcare. Be sure to read the plan document carefully to make sure \\n\\nthat the plan meets your healthcare needs. \\n\\nWHAT IF I HAVE OTHER COVERAGE? \\n\\nCoordinating Benefits With Other Health Care Plans \\n\\nWHAT IF I HAVE OTHER COVERAGE? \\n\\nCoordinating Benefits With Other Health Care Plans \\n\\nIt may be possible to coordinate benefits with other health care plans if you have other \\n\\ncoverage. Coordinating benefits allows you to receive payments from each health plan',\n",
       "  'metadata_storage_name': 'Northwind_Standard_Benefits_Details.pdf',\n",
       "  '@search.score': 0.02309836819767952,\n",
       "  '@search.reranker_score': 2.903186321258545,\n",
       "  '@search.highlights': None,\n",
       "  '@search.captions': None},\n",
       " {'content': 'preventive care services, at the rates specified in the plan documents. \\n\\n\\n\\nThe plan pays for covered services only after the member has met their annual deductible. \\n\\nThe plan pays for covered services after the member has met the annual deductible, up to \\n\\nthe maximum out-of-pocket limit. The maximum out-of-pocket limit will be specified in the \\n\\nplan documents. For covered services, Northwind Health Plus pays either a percentage of \\n\\nthe cost or a fixed dollar amount, whichever is less.  \\n\\nNorthwind Health Plus also pays for services that are not listed in the plan documents, if the \\n\\nhealth care provider determines that such services are medically necessary. This includes \\n\\nservices that are not covered under the plan, such as experimental treatments and services \\n\\nfor cosmetic purposes.  \\n\\nNorthwind Health Plus also pays for emergency services, both in-network and out-of-\\n\\nnetwork. To be eligible for coverage, the emergency must meet certain criteria, as specified \\n\\nin the plan documents. If the emergency services do not meet the criteria, the member may \\n\\nbe responsible for the full cost of the services.  \\n\\nIn addition, Northwind Health Plus may pay for services that are not covered under the \\n\\nplan, if the health care provider determines that such services are medically necessary. The \\n\\nplan will pay for these services at the rates specified in the plan documents.  \\n\\nIt is important to note that all payments are subject to the terms and conditions of the plan, \\n\\nincluding any applicable copayments, coinsurance, and deductible amounts. Members \\n\\nshould always check with their health care provider to determine if a service is covered \\n\\nunder the plan and the amount that will be paid for the service.  \\n\\nIt is also important to note that Northwind Health Plus does not pay for any services that \\n\\nare not medically necessary. Any services that are deemed to be for cosmetic purposes,',\n",
       "  'metadata_storage_name': 'Northwind_Health_Plus_Benefits_Details.pdf',\n",
       "  '@search.score': 0.012820512987673283,\n",
       "  '@search.reranker_score': 2.5313265323638916,\n",
       "  '@search.highlights': None,\n",
       "  '@search.captions': None},\n",
       " {'content': 'not a substitute for health insurance. You \\n\\nshould still maintain health insurance coverage through an employer, a private plan, or a \\n\\ngovernment-sponsored plan. The Northwind Health Plus plan is intended to supplement the \\n\\ncoverage you have from other sources.  \\n\\nTips for Making the Most of Your Plan  \\n\\n• Make sure you understand your plan documents and know what is covered and what is \\n\\nexcluded.  \\n\\n• Take advantage of preventive care services, such as check-ups and screenings, as these \\n\\nwill help you stay healthy and avoid more costly treatments down the road.  \\n\\n• Consider signing up for the Northwind Health Plus online portal, which allows you to view \\n\\nyour health plan information, make payments, and access other resources.  \\n\\n• Take advantage of Northwind Health’s 24/7 nurse hotline, which can provide you with \\n\\nmedical advice and other assistance.  \\n\\n• Use your plan’s in-network providers whenever possible, as this will help you save money.  \\n\\n• Utilize Northwind Health’s online pharmacy service, which allows you to easily order and \\n\\nmanage your prescriptions without leaving home.  \\n\\n• When you receive health care services, make sure to check that the provider is in-network \\n\\nand that the services are covered by your plan.  \\n\\n• Keep track of your claims and other plan information, as this will help you to better \\n\\nunderstand your plan and stay on top of your health care expenses.  \\n\\n\\n\\n• Utilize Northwind Health’s customer service to answer any questions or address any \\n\\nconcerns you may have about your plan. \\n\\nHealthcare Providers - Independent Contractors \\n\\nOTHER INFORMATION ABOUT THIS PLAN \\n\\nHealthcare Providers - Independent Contractors \\n\\nThe Northwind Health Plus plan includes coverage for healthcare services provided by \\n\\nindependent contractors. This means that services provided by independent contractors \\n\\nmay be covered under the Northwind Health Plus plan, provided that the service is \\n\\nmedically necessary.',\n",
       "  'metadata_storage_name': 'Northwind_Health_Plus_Benefits_Details.pdf',\n",
       "  '@search.score': 0.025837769731879234,\n",
       "  '@search.reranker_score': 2.517922878265381,\n",
       "  '@search.highlights': None,\n",
       "  '@search.captions': None},\n",
       " {'content': \"for medical services. Northwind Health Plus offers coverage for hospital stays, \\ndoctor visits, lab tests, and X-rays. Northwind Standard only offers coverage for doctor visits and lab \\ntests. \\n\\nNorthwind Health Plus is a comprehensive plan that offers more coverage than Northwind Standard. \\nNorthwind Health Plus offers coverage for emergency services, mental health and substance abuse \\ncoverage, and out-of-network services, while Northwind Standard does not. Northwind Health Plus also \\n\\n\\n\\noffers a wider range of prescription drug coverage than Northwind Standard. Both plans offer coverage \\nfor vision and dental services, as well as medical services.  \\n\\nCost Comparison\\nContoso Electronics deducts the employee's portion of the healthcare cost from each paycheck. This \\nmeans that the cost of the health insurance will be spread out over the course of the year, rather \\nthan being paid in one lump sum. The employee's portion of the cost will be calculated based on the \\nselected health plan and the number of people covered by the insurance. The table below shows a \\ncost comparison between the different health plans offered by Contoso Electronics:\\n\\nNext Steps \\nWe hope that this information has been helpful in understanding the differences between Northwind \\nHealth Plus and Northwind Standard. We are confident that you will find the right plan for you and \\nyour family. Thank you for choosing Contoso Electronics!\",\n",
       "  'metadata_storage_name': 'Benefit_Options.pdf',\n",
       "  '@search.score': 0.016393441706895828,\n",
       "  '@search.reranker_score': 2.4478399753570557,\n",
       "  '@search.highlights': None,\n",
       "  '@search.captions': None},\n",
       " {'content': \"accurate and complete information to the review team. \\n\\n\\n\\n• If your coverage is denied, talk to your doctor about appealing the decision. \\n\\n• If you are considering a service or medication that is not covered by Northwind Health \\n\\nPlus, ask your doctor about other options that may be available. \\n\\nPersonal Health Support Programs \\n\\nCARE MANAGEMENT \\n\\nNorthwind Health Plus offers a number of personal health support programs to help \\n\\nmembers stay healthy and manage their health care costs. Through this program, members \\n\\ncan access a range of services, programs, and benefits including:  \\n\\nCase Management: Northwind Health Plus provides a case management program that \\n\\nconnects members with a team of health professionals, depending on the individual’s needs. \\n\\nThese professionals will help assess the member's health situation, develop a plan of care, \\n\\ncoordinate care and resources, and provide support and education. \\n\\nDisease Management: Northwind Health Plus offers disease management programs for \\n\\nmembers with certain chronic conditions. These programs provide members with support, \\n\\ninformation, and resources about their conditions, as well as assistance in managing their \\n\\nhealth care. \\n\\nWellness Programs: Northwind Health Plus provides wellness programs to help members \\n\\nstay healthy and manage their health care costs. These programs include programs to help \\n\\nmembers quit smoking, manage stress, and improve their overall health and well-being. \\n\\nExceptions: \\n\\n•Members must be enrolled in Northwind Health Plus to be eligible for these programs. \\n\\n•These programs may not be available in all areas. \\n\\n•Some services may not be covered by Northwind Health Plus. \\n\\nTips: \\n\\n•Take advantage of the services and programs offered through Northwind Health Plus. \\n\\n•Talk to your doctor or other health care provider about your health and any treatments \\n\\nthat may be available. \\n\\n•Take an active role in your health care. Ask questions and be informed about your health\",\n",
       "  'metadata_storage_name': 'Northwind_Health_Plus_Benefits_Details.pdf',\n",
       "  '@search.score': 0.015625,\n",
       "  '@search.reranker_score': 2.415670871734619,\n",
       "  '@search.highlights': None,\n",
       "  '@search.captions': None}]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "chat_thread.get_last_message_sources()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
